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(57) ABSTRACT 

A control panel/display subsystem acts as a device portal for 
interacting with multiple devices interconnected via a 
dynamic local network. Display content and the human- 
machine interface (HMI) implemented using the display 
subsystem automatically reconfigures itself when new 
devices are added to the vehicle network. An interface 
specifier enabling each new device to work with the device 
portal is obtained either from a local archive or a remote 
archive via connection with a remote network. 

18 Claims, 3 Drawing Sheets 
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RECONFIGURABLE DISPLAY 
ARCHITECTURE WITH SPONTANEOUS 
RECONFIGURATION 

BACKGROUND OF THE INVENTION 

The present invention relates in general to a reconfig- 
urable display/control panel for controlling various elec- 
tronic accessories, and more specifically to an architecture 
for reconfigurable displays and an overall network for 
spontaneously interconnecting the displays with various 
electronic accessories or devices in a manner which auto- 
matically reconfigures menu elements shown on the recon- 
figurable display to interact with each electronic accessory. 

Reconfigurable displays are used in automotive vehicles 
in order to control a plurality of electronic accessories from 
a single control panel. Such a system reduces cost, saves 
space on the vehicle instrument panel, and makes the 
electronic accessories easier to control. A reconfigurable 
display includes a generic graphic display surface, such as a 
dot matrix, and a collection of "soft keys" (i.e., program- 
mable buttons). The function of each key is dynamically 
reconfigured via software to allow access to all the available 
functions or the accessories, typically using a menu struc- 
ture. A typical reconfigurable display subsystem may also 
include a number of "hard keys", buttons that provide instant 
access to frequently used functions (e.g., navigation, climate 
control, audio players, etc.). 

Because of their generic, reusable nature, reconfigurable 
automotive displays have facilitated an increase in the 
number of features that are made available to the user. 
Consumers are demanding ever-greater functionality from 
their electronic accessories, while product design cycles of 
the accessories are simultaneously becoming shorter. Thus, 
it becomes a major challenge for manufacturers to provide 
new and innovative system architectures while delivering 
high content, high quality products and features at a reason- 
able cost. 

First generation automotive reconfigurable display sys- 
tems utilize embedded architectures that build specific fea- 
ture content into the display design that cannot be altered or 
augmented after the design is implemented. All supported 
features must be identified at the time of initial design. While 
this approach provides high performance and low cost, it 
lacks flexibility. 

Second generation automotive reconfigurable display sys- 
tems utilize a personal computer (PQ type of architecture, 
such as the AutoPC platform. Such systems enable incre- 
mental feature deployment, wherein new features can be 
integrated seamlessly with those already present. However, 
such feature deployment is essentially a static model since 
installed software applications occupy a percentage of the 
display resources at all times. Thus, it is distinctly possible 
that system resources could be inadvertently depleted during 
installation of a new feature. Furthermore, such customiza- 
tion requires installation skills on the part of the users (not 
just system developers and integrators), which limits the 
utility of such customization for a significant percentage of 
customers. Such systems are not truly "plug-and-play" since 
a manual installation procedure is required. 

SUMMARY OF THE INVENTION 

The present invention has the advantage of providing a 
reconfigurable display architecture in which a human- 
machine interface (HMI) is dynamically constructed in 
response to the electronic accessories which are present in 
the system. 
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In one aspect of the invention, an electronic accessory 
display/control system is provided for a transportation 
vehicle. A reconfigurable control panel has a visual display 
for displaying menu items for an electronic accessory and 

5 has at least one control actuator. A human-machine interface 
controller is coupled to the reconfigurable control panel and 
includes a local archive for storing a plurality of interface 
specifiers. Each specifier defines interaction between the 
reconfigurable control panel and a respective electronic 

10 accessory for performing operations via the menu items 
using a predetermined communications protocol. The sys- 
tem includes an expandable interconnection link for cou- 
pling compatible electronic accessories with the human- 
machine interface controller. A wireless transceiver is 

15 provided for accessing a remote archive of interface speci- , 
fiers. The remote archive includes interface specifiers each 
adapted for a corresponding combination of a particular 
electronic accessory and a particular reconfigurable control 
panel. The human-machine interface controller responds to 

20 a coupling of an electronic accessory to the expandable 
interconnection link by checking the local archive for pres- 
ence of a desired interface specifier corresponding to the 
electronic accessory and the reconfigurable control panel. If 
the desired interface specifier is not present in the local 

25 archive, then the wireless transceiver is activated to auto- 
matically obtain the desired interface specifier from the 
remote archive. 

BRIEF DESCRIPTION OF THE DRAWINGS 

30 FIG. 1 is a front, plan view of a reconfigurable display 
employing the present invention. 

FIG. 2 is a schematic diagram showing the overall net- 
work system of the present invention. 
3S FIG. 3 is a block diagram showing the interaction of 
software objects for forming a human-machine interface and 
its interaction with the reconfigurable display. 

FIG. 4 illustrates the main tasks executed when a new 
device is joined into the vehicle network. 

40 DETAILED DESCRIPTION OF PREFERRED 
EMBODIMENTS 

The present invention creates a device portal within a 
network architecture having a dynamically constructed 

45 human-machine interface (HMI). A control panel/display 
subsystem includes a collection of hard and soft controls and 
is made available as a network resource on a dynamic local 
network. The display subsystem of the device portal may 
include standard embedded features such as an audio tuner 

50 or CD player, but its main purpose is to be dynamically 
reconfigurable to interact with other network resources via a 
collection of standard protocols. These other network 
resources include devices such as a navigation system, 
cellular phone, audio player, a palm-size PC, or any other; 

55 device employing an HMI in the vehicle. These devices need 
not be present in the network at all times. Using Java/Jini 
technology or similar technology, a dynamic network can be 
constructed which allows automatic installation of devices 
into the network. 

60 Referring to FIG. 1, a control panel/display subsystem 10 
includes a multi-element graphical display 11. A plurality of 
push buttons 12-17 provide soft keys for accessing func- 
tions as identified by graphic/text labels displayed on display 
screen 11. Hard controls include a knob 18 which is pressed 

65 to control system power and can be rotated to control audio 
volume. An arrow pad 19 is used to navigate through menus 
displayed on display screen 11. An enter button 20 and a 
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back button 21 are also used to navigate through menu 
screens. A plurality of shortcut buttons 22-26 are provided 
to create shortcuts to menu screens for device functionality 
embedded in subsystem 10 (e.g., CD player or AM/FM 
radio). 

An HMI for a particular electronic accessory device 
includes graphical display elements to identify the device 
and its available features. A menu screen for each accessory 
device includes labels to be displayed associated with par- 
ticular soft keys 12-17 to identify controllable functions of 
the device. For the electronic accessory of a cellular phone, 
the soft keys may be associated with cellular phone func- 
tions of accessing memory locations, initiating a call, ending 
a call, or other functions performable by the phone. Display- 
screen 11 can also be used to displ ay event information as 
commun icated from the cellula r pfione. such as connection 
statttsrtfuration of call, and other information communicated 
by the cellular phone to display subsystem 10. 

The use of the reconfigurable display subsystem as a 
device portal in a dynamic local network is shown in FIG. 
2. Reconfigurable display subsystem 30 includes a display 
screen 31 and hard and soft keys 32. Display subsystem 30 
may also receive input commands from a voice recognition 
unit 33. An HMI controller 34 resides in display subsystem 
30 and controls graphical display screen 31, monitors keys 
32, accepts input from voice recognition (VR) unit 33, and 
interfaces with devices on the dynamic local network 36. A 
memory 35 stores interface specifier s (i.e., drivers) used by 
controller .54 to drive graphic display screen 31 and to 
communicate with the various electronic accessory devices 
on dynamic local network 36. 

Each particular display subsystem ("device portal") 
design is uniquely identified by a type identifier. Thus, 
display subsystem 30 is identified as type 0, while additional 
display subsystems which may be connected to the local 
dynamic network 36 have different identifiers such as type 
1 for a display subsystem 37 and type n for a display 
subsystem 38. 

Dynamic local network 36 includes a collection of soft- 
ware and communication specifications and standard proto- 
cols for hardware interconnection. Examples of such a 
system are Jini by Sun Microsystems, Inc., JetSend by 
Hewlett-Packard, and Bluetooth by the Bluetooth Special 
Interest Group. System resources such as reconfigurable 
display subsystems, electronic accessories or other compo- 
nents can join the network automatically once they are 
connected to it. Network 36 recognizes the coupling of a 
new device to the network and interacts with all the network 
resources as appropriate to enable operation of the new 
device within the network. Examples of electronic accesso- 
ries connected to the network in FIG. 2 include a cellular 
phone 40, an MP3 audio player 41, and a pal m-sized PC o r 
personal digital assistant (PDA) 42. Once connected to the 
network 3b, these accessories will communicate core func- 
tionality control signals and messages with a particular 
display subsystem. Thus, the accessory and the reconfig- 
urable display subsystem will exc hange messages concerrj - 
ing control.actions and state changes or events out would not 
Include specific messages on how to display messages" or 
how the display is to be driven. 

Eaclrdevice 40-42 includes a unique device type identi- 
fier. Each device type may interact with a predetermined 
reconfigurable display type using an interface specifier 
developed for the combination of device and display sub- 
system. Thus, when HMI controller 34 detects the presence 
of a new electronic accessory, it determines the device type 
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for the accessory and checks whether it currently has an 
interface specifier to support interaction with the device 
stored in memory 35. If the desired interface specifier is 
present, then HMI controller 34 can communicate core 

5 functionality messages between the reconfigurable display 
and the accessory device. If an appropriate interface speci- 
fier is not already contained in memory 35, then HMI 
controller 34 takes steps to retrieve an appropriate interface 
specifier, if possible. 

10 A memory in each reconfigurable display subsyste m 
provides X local archive for storing a plurality of inter lace* 
specifiers each' ot' which defines interaction between ine 
reconfigurable display subsystem and a respective electronic 
accessory. Whenever an additional interface specifier must 

25 be retrieved, it may preferably be obtained using a universal 
resource locator (URL) of a server that contains a further 
collection of HMI interface specifiers. Such a server may 
also be a local archive in the vehicle directly connected to 
dynamic local network 36 as shown by a server 43 in FIG. 

20 2. Server 43 is a local server containing a first group of HMI 
mterTaMTspecifiers 44 corresponding to the configurable 
display type zero. Additional HMI interface specifiers are 
stored in other groups for other reconfigurable display types 
as shown. For each reconfigurable display type, a plurality 

25 of interface specifiers are stored as indexed by device type. 
Server 43 may be constructed with some interface specifiers 
contained in a read-only memory (ROM) in order to provide 
a fixed set of interface specifiers for a known set of elec- 
tronic accessories which are expected to be utilized in a 

30 particular vehicle. In addition, re-writeable memory may 
also be included for subsequent storage of interface speci- 
fiers for other device types in order to provide flexibility for 
growth. 

In order to accommodate electronic accessory devices not 

35 included in local server 43, the present invention also 
provides access to a remote archive web server outside the 
vehicle. Thus, a wireless modem 45 is interconnected with 
dynamic local network 36 and can be used to communicate 
with a remote wireless modem 46 which is connected to a 

40 remote web server 47 containing additional interface speci- 
fiers in a remote archive. Remote server 47 may be con- 
nected to the world-wide web or internet and wireless 
modem 46 may be connected to an internet service provider 
(ISP), for example. The URL address for remote server 47 

45 may be a predetermined address as defined by convention 
and stored in either local server 44 or HMI controller 34, for 
example. Preferably, the URL address of an remote archive 
may be obtained directly from each accessory device itself. 
Thus, cellular phone 40 stores a remote archive address 

50 providing the URL to the other resources on dynamic local 
network 36. Thus, cellular phone 40 stores URL of 
www.visteon.com/hmicode where an appropriate interface 
specifier corresponding to cellular telephone device type 0 
and a plurality of reconfigurable display types are stored. 

55 Thus, as new electronic accessory devices are developed, 
interface specifiers can also be developed in order to inter- 
face the new device with the existing reconfigurable display 
types. The location from where these interface specifiers can 
be retrieved is stored in the new accessory device, thus 

60 making the device compatible with all reconfigurable dis- 
plays of those types. 

FIG. 3 shows the various software elements required in 
the system of FIG. 2, including software elements compris- 
ing the HMI itself, (i.e., the interface between the display 

65 subsystem and the accessory device and software for driving 
the display). Thus, a human-machine interface software 
element 50 provides application objects for a particular 



06/09/2004, EAST Version: 1.4.1 



US 6,559, 

5 

accessory such as climate control, radio tuner, wireless 
information service, e-mail, cellular phone, audio, CD 
player, or others. These objects interact with other objects in 
an HMI widgets component library 51 including such 
objects as button metaphor (i.e., button icon and identifica- 5 
tion of corresponding soft key), list box, window, text box, 
time, and others. These objects interact with graphics primi- 
tives 52 providing a graphics device interface. These graph- 
ics primitives define vector shapes, raster elements, perform 
window management, and provide graphic attributes. These 10 
primitives interact with software for display frame buffer 53 
for managing display activation such as target selection, 
visibility selection, and drawing mode. 

More specifically, an interface specifier which would be 
downloaded from either a local or a remote archive contains 15 
compiled software class objects that collectively implement 
an application specific HMI for the unique display driver/ 
accessory device combination.. In a Java implementation, 
these objects-will be precompiled from Java source code into. 
Java bytecodes which are the. instructions that run on the 20 
Java Virtual machine (JVM). Some of these downloaded 
objects implement an overall HMI for the specific class of 
^application of the accessory device, such as cellulaTpriSrle, 
compact disc player, or address book. Some of the other 
downloaded objects are generic (i.e., application 25 
independent) and can be applied to a wide range of appli- 
cations. These generic, reusable components or widgets may 
typically already reside in the display subsystem, but may be 
included in a downloaded interface specifier for complete- 
ness and flexibility to use display subsystems not already 30 
containing the widgets. Downloading may include the capa- 
bility to identify objects already residing in the display 
subsystem and then only downloading objects which are in 
fact needed. 

The behavior of a particular HMI is embedded in the 35 
collection of class objects within the interface specifier and 
include four main functional areas: 1) processing user input 
events, 2) processing device events, 3) rendering graphic 
displays, and 4) sending commands to devices. 

User input events are generated when a user manipulates 
a control actuator of the display subsystem, such as pressing 
a button, issuing a voice command, manipulating the point- 
ing device such as a mouse or track ball, or otherwise 
initiates a control action. Physical device events are repre- 45 
sented by software abstractions and are reported to the 
accessory device via the dynamic local network. Examples 
of physical device events include button pressed, button 
held, button released, switch closed, switch opened, pointing 
device position change, pointing device pressed, pointing 5Q 
device held, and pointing device released. The control 
actuator being used may be physically contained on the 
display subsystem or may be remotely connected to the 
display subsystem, such as a pointing device mounted on a 
steering wheel in the vehicle. 55 

Objects for processing device events provide notification 
to the display subsystem of state changes occurring within 
the electronic accessory device. Specific device events 
depend upon the functionality of the particular electronic 
accessory device. For an electronic accessory device pro- 60 
viding navigation features, examples of device events 
include notification of a pending route maneuver, vehicle 
approaching destination, vehicle off- route, and others. 

With regard to software objects to render graphics on the 
visual display, these objects respond to user input or device 65 
or system events (e.g., power-up initialization) to initiate and 
execute all required rendering operations. The HMI inter- 
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face specifier embeds the knowledge of how to display the 
information, including font size, screen location, number of 
digits, size and shape of graphic elements, timing for ani- 
mated components, etc. 

The software objects include those that send predeter- 
mined commands to the electronic accessory devices in 
order to enable the user to control and monitor the devices. 
Some typical examples include incrementing or decrement- 
ing a CD track selection, selecting an FM radio pre-set, 
adjusting the time-of-day clock, setting climate control 
temperature or fan speed, dialing a phone number, and 
others. The specific implementation of the user interface 
specifier can enable these actions in many different ways. 
These objects correlate user input events with the corre- 
sponding control action. 

A preferred method of the present invention will be 
described in connection with FIG. 4. The sequence of steps 
shows interaction between an accessory device 60, a recon- 
figurable display subsystem 61, and a server 62 containing 
an archive of interface specifiers. Device 60 is a pluggable 
device such as a cellular phone, palm-size PC, or MP3 
player and is identified via a device type identifier which is 
embedded into the device's persistent storage, such as ROM 
or FLASH. Persistent storage also contains a universal 
resource locator (URL) that indicates the location of an HMI 
server containing interface specifiers corresponding to the 
device. Display subsystem 61 likewise has its own display 
type identifier embedded into persistent storage. Server 62 
may be a local server or a world-wide web server containing 
collections of HMI interface specifiers for the various dis- 
play subsystem and accessory device combinations. The 
server processes requests for interface specifiers which 
include the device type identifier and the display type 
identifier. The first phase of the process is registration. In 
registration, device 60 communicates its presence to 
resources on the dynamic local network and provides its 
device type identifier and the URL of a server containing a 
collection of interface specifiers corresponding to the 
device. The second phase of the process is an HMI check in 
which the display subsystem 61 checks to see if it already 
has an interface specifier supporting the accessory device 
represented by the device type identifier. If a supporting 
interface specifier is already present, then the device auto- 
matically begins to use the display subsystem as a device 
portal. 

If a supporting interface specifier is not present, then the 
third phase of the process is entered which comprises an 
HMI request. This phase includes attempting to connect to 
the server specified by the device. If connection is 
successful, the display subsystem communicates the device 
type identifier and the display type identifier to the server. 

HMI report is the fourth phase of the process. Once the 
server successfully receives a request it checks its arcBIve of 
HMI interface specifiers for a match as. defined by the 
display type identifier and device type identifier. If a match 
is found, it returns the interface specifier packaged as a Java 
archive (JAR) file, for example. If not, it sends a message 
specifying that no interface specifier is available for the 
device/display combination. In the clean-up phase, display 
subsystem 61 responds to the information obtained from the 
server. If the server i ndicated that no interface specifier w as 
available, then "the display subsystem indicates an errors 
condition to the user. If a JAR file was successfully returned, 
then the display subsystem extracts and installs the new 
software objects automatically to allow the display sub- 
system to act as a device portal for the installed device. 

In order to minimize the amount of memory required for 
storing interface specifiers, and to simultaneously reduce 
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download times required, a memory manager or caching 2. The electronic accessory display and control system of 

technique is used for storing interface specifiers. Preferably, claim 1 wherein said desired interface specifier assigns a 

a memory in any individual subsystem is large enough to predetermined function to said control actuator. 

hold interface specifiers to service the complete maximum 3. The electronic accessory display and control system of 

load of accessory devices that may be in operation using the 5 c i ami i wherein said expandable interconnection link is 

display subsystem at any one time. Since some devices may comprised of a dynamic local network. 

be disconnected after having been used in the system, some 4 ^ e i ectr onic accessory display and control system of 

interface specifiers may be present in memory in a display daim 3 wherein ^ d ic local network is compriscd of 

subsystem for which the original accessory device is no a netWQrk and wherein nce of new electronic 

longer present in the network. According to the present 10 . - A , 4 , *• h ■ 

. & , . ■ • . r -a - ■ accessories is detected automatically in response to a wire- 

mvention, any such inactive interface specifier is given a j esg ^ s - a j 

lower priority man active interface specifiers. Furthermore, «%■!_?! ■' j-t ^ t c 

the looger an interface specifier has been inactive, the lower , 5 ' ™ e el^onic accessory display and control system of 

priority it is given relative to other inactive interface sped- cla ™ 1 wherein said compatible electronic accessories each 

fiers. If the amount of remaining free memory becomes is provide to said human-machine interfaa controller a unique 

restricted, the HMI controller deletes the lowest priority device identifier and a remote network address for said 

interface specifiers to make room for additional interface rcD ?°i^ arcmve - 

specifiers for other accessory devices that have joined the , ? ™ e dectamic accessory display and control system of 

network. Priority assignments may also take into account d ¥ m 1 wh ? rem mterface t^Bta are comprised of 

file size and/or frequency of use so as to minimize overall 20 _ * re 0D J ects ' 

downloading to restore interface specifiers that have been , T ™ e electronic accessory display and control system of 

de l eted claim 6 wherein said software objects include objects to 

T * . - . 4 . process user events initiated by said control actuator. 

Local server 43 in FIG. 2 may also contain a memory o tu 1 . _ i. , , . , # f 

- . 1 . .... ,, , . J 8. The electronic accessory display and control system of 

manager 48 for similarly managing prioritizing and/or dele- , . c u -a <■* u- . ■ 1 j L- * * 

r TT »« • r . % 1 „ claim 6 wherem said software objects include objects to 

tion of HMI interface specifiers to most effectively use 25 . J 

A t process device events corresponding to a state change within 

storage space in server 43. -jr.- 

& r said electronic accessory. 

With regard to the dynamic local network, this may take 9 ^ electronic accessory display and control system of 

the form of an RF-wireless network using the Bluetooth claim 6 wherem said software objects include objects to 

specification, for example. When a wireless-capable acces- render graphics on said visual fo^y. 

sory comes within communication distance of the dynamic 10 . ^ eIectronic accessory display and control system of 

local network, this is detected by means of a wireless polling claim 6 wherein said soft^ objects include objects t0 

signal exchanged between the local network and the new send predetermined commands to said electronic accessory, 

device. Based on a response to the polling signal, the devices u ^ electronic accessory display and control system of 

exchange network messages to establish the new device as daim t including said electronic accessory, and 

a resource available to devices on the local network. wherein ^ electronic accessory is comprised of a portable 

What is claimed is: computing device. 

1. An electronic accessory display/control system for a n electronic accessory display and control system of 

transportation vehicle, comprising: claim x including said electronic accessory, and 

a reconfigurable control panel having a visual display for wherein said electronic accessory is comprised of a mobile 

displaying menu items for an electronic accessory and communication device. 

having at least one control actuator; 13, The electronic accessory display and control system of 

a human-machine interface controller coupled to said claim 1 wherein said control actuator is comprised of a push 

reconfigurable control panel and including a local button switch. 

archive for storing a plurality of interface specifiers, 45 14. The electronic accessory display and control system of 

each specifier defining interaction between said recon- claim 1 wherein said control actuator is comprised of a 

figurable control panel and a respective electronic speech recognition unit. 

accessory for performing operations via said menu 15. The electronic accessory display and control system of 

items using a predetermined communications protocol; claim 1 further comprising a memory manager for priori- 

an expandable interconnection link for coupling compat- 50 tizing interface specifiers stored in said local archive and 

ible electronic accessories with said human-machine deleting interface specifiers of lower priority when said local 

interface controller; and archive becomes full. 

a wireless transceiver for accessing a remote archive of 16 * A method of operating an electronic accessory display 

interface specifiers, wherein said remote archive and control system for a transportation vehicle, said system 

includes interface specifiers each adapted for a corre- 55 ^eluding a reconfigurable control panel, a human-machine 

sponding combination of a particular electronic acces- interface controller, an expandable interconnection link for 

sory and a particular reconfigurable control panel; compatible electronic accessories, and a wireless 

wherein said human-machine interface controller transceiver, said method comprising the steps of: 

responds to a coupling of an electronic accessory to storing a plurality of interface specifiers in a local archive, 

said expandable interconnection link by checking said 60 eacn interface specifier defining interaction between 

local archive for presence of a desired interface speci- said reconfigurable control panel and a respective elec- 

fier corresponding to said electronic accessory and said tronic accessory for performing operations via menu 

reconfigurable control panel, and if said desired inter- items usiag a predetermined protocol for said recon - 

face specifier is not present in said local archive then figurable control panel; 

activating said wireless transceiver to automatically 65 coupling one of said compatible electronic accessories 

obtain said desired interface specifier from said remote with said human-machine interface controller via said 

archive. expandable interconnection link; 
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checking, in response to said coupling, said local archive 
for presence of a desired interface specifier correspond- 
ing to said compatible electronic accessory and said 
reconfigurable control panel; and 

if said desired interface specifier is not present in said 5 
local archive, then activating said wireless transceiver 
to automatically obtain said desired interface specifier 
from a remote archive containing an interface specifier 
adapted for combination of said compatible electronic 
accessory and said reconfigurable control panel. 
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17. The method of claim 16 wherein said coupled com- 
patible electronic accessory stores a remote network address 
of said remote archive. 

18. The method of claim 16 further comprising the steps 

of: 

prioritizing said interface specifiers stored in said local 
archive; and 

deleting an interface specifier of lower priority when said 
local archive becomes full. 

* * * * * 
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